Image sensor, image capturing apparatus and image processing apparatus

ABSTRACT

An image sensor comprises: a pixel region including a plurality of microlenses arranged in a matrix, and a plurality of photoelectric conversion portions provided for each of the microlenses; a plurality of amplifiers that apply a plurality of different gains to signals output from the pixel region; and a scanning circuit that scans the pixel region so that a partial signal and an added signal are read out, the partial signal being a signal from some of the plurality of photoelectric conversion portions, and the added signal being a signal obtained by adding the signals from the plurality of photoelectric conversion portions.

BACKGROUND OF THE INVENTION Field of the Invention

The present invention relates to an image sensor, an image capturingapparatus and an image processing apparatus.

Description of the Related Art

Recently, techniques have been proposed in which an image sensor notonly simply outputs image signals obtained through photoelectricconversion in pixels, but also expands the dynamic range, outputsinformation indicating the distance to an object, and the like, forexample. Japanese Patent Laid-Open No. 2005-175517 proposes a techniquethat provides a function for switching the input capacitances ofamplifying circuits provided in each column of an image sensor, wherethe gain is switched in accordance with a signal level. With aconfiguration that switches the gain as described in Japanese PatentLaid-Open No. 2005-175517, low-gain and high-gain image signals areoutput and synthesized through image processing carried out at a laterstage, which makes it possible to create a high-dynamic range andlow-noise image signal.

Meanwhile, a focus detection method using what is known as an on-imagingplane phase difference detection method (on-imaging plane phasedifference AF) has been proposed, in which a pair of images havingparallax is read out from the image sensor and the focus is detectedusing a phase difference detection method. An image capturing apparatusin which a pair of photoelectric conversion portions are provided foreach of microlenses constituting a microlens array arrangedtwo-dimensionally can be given as an example of an image capturingapparatus outputting signals that can be used in a focus detectionmethod employing the on-imaging plane phase difference detection method.Japanese Patent Laid-Open No. 2001-83407 proposes an image capturingapparatus in which whether or not to add signals output from each ofpairs of photoelectric conversion portions, on which light is incidentthrough a single microlens, can be controlled as desired for each pairof photoelectric conversion portions.

However, the way in which high-gain image signals and low-gain imagesignals for expanding the dynamic range are read out in Japanese PatentLaid-Open No. 2005-175517 is different from the way in which the imagesignals for phase difference detection are read out in Japanese PatentLaid-Open No. 2001-83407, and those signals cannot be read out in thesame frame.

Additionally, consider a situation where, in a single frame, driving forreading out image signals for expanding the dynamic range and drivingfor reading out image signals for phase difference detection areswitched from readout row to readout row of the image sensor in order tomaintain a high framerate. In this case, the dynamic range cannot beexpanded in rows where the image signals for phase difference detectionare read out.

Although it is conceivable to apply high gain and low gain for expandingthe dynamic range as per Japanese Patent Laid-Open No. 2005-175517 tothe image signals for phase difference detection according to JapanesePatent Laid-Open No. 2001-83407 and read out the signals, the followingproblem can arise in such a case. With long exposures or inhigh-temperature environments, factors such as an increase in darkcurrent components, changes in wiring resistances, and the like maycause differences in black levels between the image signals to whichdifferent gains have been applied, resulting in an offset shift. Ifimage data obtained through expanding dynamic range synthesizing (HDRsynthesizing) and signals for phase difference detection are used in astate where the offset is shifted in this manner, on-imaging plane phasedifference AF cannot be applied appropriately.

SUMMARY OF THE INVENTION

Having been made in consideration of the above situation, the presentinvention reads out, in a short time, image signals necessary for focusdetection and/or dynamic range expansion in accordance with the subjectbeing shot and the application of the read-out image signals.

Additionally, the present invention makes it possible to obtain an imagesignal and a suitable pupil division signal when carrying out dynamicrange expansion synthesis.

According to the present invention, provided is an image sensorcomprising: a pixel region including a plurality of microlenses arrangedin a matrix, and a plurality of photoelectric conversion portionsprovided for each of the microlenses; a plurality of amplifiers thatapply a plurality of different gains to signals output from the pixelregion; and a scanning circuit that scans the pixel region so that apartial signal and an added signal are read out, the partial signalbeing a signal from some of the plurality of photoelectric conversionportions, and the added signal being a signal obtained by adding thesignals from the plurality of photoelectric conversion portions.

Further, according to the present invention, provided is an imagecapturing apparatus comprising: an image sensor, the image sensorincluding a pixel region having a plurality of microlenses arranged in amatrix and a plurality of photoelectric conversion portions provided foreach of the microlenses, a plurality of amplifiers that can amplifysignals output from the pixel region using a plurality of differentgains including at least a first gain, and a scanning circuit that scansthe pixel region so that a partial signal and an added signal are readout, the partial signal being a signal from some of the plurality ofphotoelectric conversion portions, and the added signal being a signalobtained by adding the signals from the plurality of photoelectricconversion portions; a controller that controls the scanning circuit; aprocessor that expands a dynamic range using the added signal amplifiedusing the plurality of different gains; and a focus detection circuitthat carries out phase difference focus detection using the partialsignal and the added signal both of which are amplified using the firstgain.

Furthermore, according to the present invention, provided is an imagesensor comprising: a pixel region including a plurality of microlensesarranged in a matrix, and a plurality of photoelectric conversionportions provided for each of the microlenses; a plurality of amplifiersthat apply a plurality of different gains to signals output from thepixel region; and a scanning circuit that scans the pixel region so thatsignals are read out in parallel from each of the plurality ofphotoelectric conversion portions.

Further, according to the present invention, provided is an imagecapturing apparatus comprising: an image sensor, the image sensorincluding a pixel region having a plurality of microlenses arranged in amatrix and a plurality of photoelectric conversion portions provided foreach of the microlenses, a plurality of amplifiers that amplify signalsoutput from the pixel region using a plurality of different gainsincluding at least a first gain, and a scanning circuit that scans thepixel region so that signals are read out in parallel from each of theplurality of photoelectric conversion portions; a controller thatcontrols the scanning circuit; a processor that expands a dynamic rangeusing an added signal, the added signal obtained by adding signalsamplified using the plurality of different gains; and a focus detectioncircuit that carries out phase difference-type focus detection usingsignals output from each of the plurality of photoelectric conversionportions, amplified using the first gain.

Further, according to the present invention, provided is an imageprocessing apparatus comprising: a synthesizer that carries out dynamicrange expansion synthesis, using a first coefficient, on a plurality ofimage signals amplified at a plurality of different gains and outputfrom an image sensor, and carries out dynamic range expansion synthesis,using a second coefficient, on a plurality of first signals amplified atthe plurality of different gains using pupil-divided partial signals andoutput from the image sensor; and a processor that finds a second signalby subtracting the synthesized first signals from the synthesized imagesignals corresponding to the synthesized first signals, wherein thesynthesizer determines the second coefficient on the basis of the firstcoefficient.

Further features of the present invention will become apparent from thefollowing description of exemplary embodiments (with reference to theattached drawings).

BRIEF DESCRIPTION OF THE DRAWINGS

The accompanying drawings, which are incorporated in and constitute apart of the specification, illustrate embodiments of the invention, andtogether with the description, serve to explain the principles of theinvention.

FIGS. 1A and 1B are diagrams schematically illustrating a configurationof an image sensor according to embodiments of the present invention.

FIG. 2A is a diagram illustrating, in detail, parts of the image sensor,from a unit pixel to an A/D circuit group.

FIG. 2B is a circuit diagram illustrating a configuration of a columnamplifier.

FIG. 3 is a timing chart illustrating control of the column amplifierwhen reading out a partial signal for phase difference detection and animage signal for dynamic range expansion, according to embodiments ofthe present invention.

FIG. 4 is a timing chart illustrating control of the column amplifierwhen reading out a partial signal for phase difference detection and animage signal when the dynamic range is not expanded, according toembodiments of the present invention.

FIG. 5 is a timing chart illustrating control of the column amplifierwhen reading out an image signal for dynamic range expansion withoutreading out a partial signal for phase difference detection, accordingto embodiments of the present invention.

FIGS. 6A and 6B are diagrams illustrating readout timing of imagesignals from the image sensor according to a first embodiment.

FIGS. 7A and 7B are diagrams illustrating the readout timing of imagesignals from the image sensor according to the first embodiment.

FIG. 8 is a block diagram illustrating an overall configuration of animage capturing apparatus according to the first embodiment.

FIGS. 9A and 9B show a flowchart illustrating readout control of theimage sensor according to the first embodiment.

FIGS. 10A and 10B are diagrams illustrating readout timing of imagesignals from the image sensor according to a second embodiment.

FIG. 11 is a block diagram illustrating an overall configuration of animage capturing apparatus according to the second embodiment.

FIGS. 12A to 12D are diagrams illustrating image data in each of blocksof the image capturing apparatus according to the second embodiment.

FIG. 13 is a block diagram illustrating an overall configuration of animage capturing apparatus according to a third embodiment.

FIG. 14 is a flowchart illustrating processing according to the thirdembodiment.

FIG. 15 is a block diagram illustrating an overall configuration of animage capturing apparatus according to a fourth embodiment.

FIG. 16 is a flowchart illustrating processing according to the fourthembodiment.

FIG. 17 is a diagram illustrating, in detail, parts of the image sensor,from a unit pixel to an A/D circuit group, according to a fifthembodiment.

FIG. 18 is a timing chart illustrating control of a column amplifierwhen reading out partial signals at high gain and low gain from twophotoelectric conversion elements in parallel, according to the fifthembodiment.

FIG. 19 is a block diagram illustrating a configuration of an imagecapturing system according to a sixth embodiment of the presentinvention.

FIGS. 20A and 20B are diagrams illustrating a concept of an HDRsynthesizing process according to the sixth embodiment.

FIG. 21 is a diagram illustrating a problem arising when the sixthembodiment is not applied.

FIG. 22 is a diagram illustrating an example of an input luminosity andan output luminosity of an HDR synthesizing unit when the sixthembodiment is applied.

FIG. 23 is a flowchart illustrating operations of the image capturingsystem according to the sixth embodiment.

FIGS. 24A and 24B are diagrams illustrating a concept of the readout ofsignals output from an image sensor according to a seventh embodiment.

FIG. 25 is a flowchart illustrating operations of an image capturingsystem according to the seventh embodiment.

DESCRIPTION OF THE EMBODIMENTS

Exemplary embodiments of the present invention will be described indetail in accordance with the accompanying drawings. The dimensions,materials, shapes and relative positions of the constituent parts shownin the embodiments should be changed as convenient depending on variousconditions and on the structure of the apparatus adapted to theinvention, and the invention is not limited to the embodiments describedherein.

First Embodiment

FIG. 1A is a diagram illustrating an example of the configuration of animage sensor equipped with an A/D converter according to embodiments ofthe present invention. A plurality of unit pixels 101, each constitutedby photodiodes for photoelectric conversion or the like, are arranged ina matrix in a pixel region 100. Each of the unit pixels 101 isconstituted by a photoelectric conversion portion A and a photoelectricconversion portion B for a single microlens 111 (described later) forthe purpose of phase difference detection, and focus detection can becarried out by finding a phase difference between image signals obtainedfrom the photoelectric conversion portion A and the photoelectricconversion portion B.

FIG. 1B is a conceptual diagram illustrating a cross-section of the unitpixel 101, where the photoelectric conversion portion A and thephotoelectric conversion portion B, each of which includes a photodiode,are provided below the single microlens 111. Each unit pixel 101includes a color filter 112. Although RGB primary color-type colorfilters provided in a Bayer pattern, in which one of R (red), G (green),and B (blue) colors corresponds to each pixel, is typically used, thecolor filters are not necessarily limited thereto.

A vertical scanning circuit 102 controls the timing for sequentiallyreading out pixel signals, which have been accumulated in thephotoelectric conversion portions A and the photoelectric conversionportions B in the pixel region 100, within a single frame period. In asingle frame period, the pixel signals are typically read out on arow-by-row basis, in sequence from the top row to the bottom row. In thepresent embodiment, control is carried out so that the vertical scanningcircuit 102 reads out a partial signal (an A signal), which is a signalfrom the photoelectric conversion portion A, and an added signal (an A+Bsignal), which is a signal obtained by adding the signals from thephotoelectric conversion portion A and the photoelectric conversionportion B, from each unit pixel 101. By reading out the signals in thismanner, the A+B signal can be used as-is as an image signal, whereas a Bsignal can be obtained by subtracting the A signal from the A+B signaland then used in focus detection using an on-imaging plane phasedifference detection method. However, it is also possible to read outonly the A+B signal if focus detection using the on-imaging plane phasedifference detection method is not to performed.

A column amplifier group 103 is constituted by a plurality of columnamplifiers, which are provided for each column in the pixel region 100,and is used for electrically amplifying the signals read out from thepixel region 100. Amplifying the signals using the column amplifiergroup 103 makes it possible to amplify the pixel signal levels withrespect to noise arising in an A/D circuit group 104 in a later stage,which substantially improves the S/N ratio. Note that the columnamplifier group 103 can amplify the signals using multiple gains, and inthe present embodiment, the dynamic range is expanded by synthesizingsignals amplified at different gains. The configuration of each columnamplifier will be described in detail later with reference to FIG. 2B.

The A/D circuit group 104 is constituted by a plurality of circuits,which are provided for each column in the pixel region 100, and convertsthe signals amplified by the column amplifier group 103 into digitalsignals. The pixel signals converted into digital signals are read outsequentially by a horizontal transfer circuit 105 and input to a signalprocessing unit 106. The signal processing unit 106 is a circuit thatprocesses signals digitally, and in addition to carrying out offsetcorrection such as FPN correction in the digital processing, is capableof performing simple gain computations through shift computations andmultiplication. The signals are output to the exterior of the imagesensor after the processing.

Memory 107 has a function for temporarily holding the A signals, the A+Bsignals, and the like read out from the pixel region 100 and processedby the column amplifier group 103, the A/D circuit group 104, and thesignal processing unit 106.

In the example illustrated in FIG. 1B, the configuration is such thatthe photoelectric conversion portion A and the photoelectric conversionportion B are provided for the single microlens 111 in each unit pixel101, but the number of photoelectric conversion portions is not limitedto two, and may be higher. The pupil division direction may be thehorizontal direction or the vertical direction, and may be a combinationof these as well. Furthermore, a plurality of pixels may be provided inwhich the opening positions of the light-receiving portions aredifferent relative to the microlenses 111. In other words, anyconfiguration may be employed in which two signals for phase differencedetection, namely the A signal and the B signal, can be obtained.Furthermore, the present invention is not limited to a configuration inwhich all pixels have a plurality of photoelectric conversion portions,and the configuration may instead be such that a pixel such as thatillustrated in FIG. 2A is provided discretely within normal pixelsconstituting the image sensor. A plurality of types of pixels, whichhave been divided using mutually-different division methods, may also beincluded within the same image sensor.

Next, the circuit configuration and the flow of signals from the unitpixel 101 to the A/D circuit group 104 will be described using FIG. 2B.A photoelectric conversion element 1101, which corresponds to thephotoelectric conversion portion A in FIG. 1B, and a photoelectricconversion element 1102, which corresponds to the photoelectricconversion portion B in FIG. 1B, share a microlens, and convert lightinto a charge through photoelectric conversion. A transfer switch 1103transfers the charge produced by the photoelectric conversion element1101 to the circuitry in the later stages, and a transfer switch 1104transfers the charge produced by the photoelectric conversion element1102 to the circuitry in the later stages. A charge holding unit 1105temporarily holds the charges transferred from the photoelectricconversion element 1101 and the photoelectric conversion element 1102when the transfer switches 1103 and 1104 are on. Accordingly, the chargeholding unit 1105 is capable of holding the charge from either thephotoelectric conversion element 1101 or the photoelectric conversionelement 1102, or a charge obtained by adding the charges from thephotoelectric conversion element 1101 and the photoelectric conversionelement 1102 together. A pixel amplifier 1106 converts the charge heldin the charge holding unit 1105 into a voltage signal, and sends thatsignal to a column amplifier 103 i in a later stage through a verticaloutput line 1113. A current control unit 1107 controls current in thevertical output line 1113.

As described above, the column amplifier group 103 illustrated in FIG.1A is constituted by a plurality of the column amplifiers 103 i, whichare provided for each column, and the column amplifier group 103amplifies the signals output through the vertical output lines 1113 andoutputs the resulting signals to the A/D circuit group 104 in a laterstage. Each of A/D circuits 104 i, which constitute the A/D circuitgroup 104, converts analog signals output from the column amplifiers 103i in the same column into digital signals.

In the A/D circuit 104 i, a digital signal converted by an A/Dconversion unit 1109 is temporarily held in memory 1110 and memory 1111.The memory 1110 holds a pixel signal read out from the photoelectricconversion element 1101 or the photoelectric conversion element 1102,and a noise signal from a readout circuit unit (which, for the sake ofsimplicity, refers to the circuitry from the charge holding unit 1105 tothe A/D conversion unit 1109). On the other hand, the memory 1111 holdsthe noise signal from the readout circuit unit. A signal obtained by asubtraction unit 1112 subtracting the data held in the memory 1111 fromthe data held in the memory 1110 is output to the horizontal transfercircuit 105 as a pixel signal.

FIG. 2B is a diagram illustrating the configuration of the columnamplifier 103 i. The column amplifier 103 i is an inverting amplifiercircuit constituted by an op-amp 207, input-side capacitors 202 and 203,and feedback capacitors 205 and 206. The connections of the capacitors202, 203, and 205 can be switched by switches 200, 201, and 204.

First, a signal input from the unit pixel 101 is accumulated in thecapacitors 202 and 203 in response to the switches 200 and 201 turningon. Then, if the signal is an image signal with proper exposure, a highgain is applied to the image signal, after which the image signal isread out, by turning the switches 201 and 204 off and the switch 200 on.Next, when reading out an image signal of a high-luminance portion, alow gain is applied to the image signal, after which the image signal isread out, by turning the switch 200 off and turning the switches 201 and204 on. In this manner, the image signals can be read out with differentgains applied thereto by using the switches to switch the capacitance ofthe capacitors. The dynamic range is then expanded by synthesizing theimage signals read out in this manner.

FIG. 3 is a timing chart illustrating control of the column amplifier103 i when reading out a partial signal for phase difference detectionand an image signal for dynamic range expansion.

First, from time t1 to t4, the gain of the column amplifier 103 i is setto a high gain by turning the switch 200 on and turning the switches 201and 204 off. In this state, the transfer switch 1103 is turned on, andthe A signal is read out, at time t2. At this time, the A signal readout at a high gain is A/D converted by the A/D circuit 104 i during theperiod from time t2 to t4.

Next, from time t4 to t5, the gain of the column amplifier 103 i is setto a low gain by turning the switch 200 off and turning the switches 201and 204 on. At this time, the A signal read out at a low gain is A/Dconverted by the A/D circuit 104 i during the period from time t4 to t5.

From time t5 to t8, the gain of the column amplifier 103 i is again setto a high gain by turning the switch 200 on and turning the switches 201and 204 off. In this state, the transfer switch 1104 is turned on, andthe B signal is read out, at time t6. The A signal and the B signal areadded in the charge holding unit 1105 and output as the A+B signal. Atthis time, the A+B signal read out at a high gain is A/D converted bythe A/D circuit 104 i during the period from time t6 to t8.

From time t8 to t9, the gain of the column amplifier 103 i is set to alow gain by turning the switch 200 off and turning the switches 201 and204 on. At this time, the A+B signal read out at a low gain is A/Dconverted by the A/D circuit 104 i during the period from time t8 to t9.

FIG. 4 is a timing chart illustrating control of the column amplifier103 i when reading out a partial signal for phase difference detectionand an image signal when the dynamic range is not expanded.

In this control, from time t11 to t16, the gain of the column amplifier103 i is set to a high gain by turning the switch 200 on and turning theswitches 201 and 204 off. In this state, the transfer switch 1103 isturned on at time t12, and the A signal is read out. At this time, the Asignal read out at a high gain is A/D converted by the A/D circuit 104 iduring the period from time t12 to t14.

Next, the transfer switch 1104 is turned on at time t14, and the Bsignal is read out. The A signal and the B signal are added in thecharge holding unit 1105 and output as the A+B signal. At this time, theA+B signal read out at a high gain is A/D converted by the A/D circuit104 i during the period from time t14 to t16.

FIG. 5 is a timing chart illustrating control of the column amplifier103 i when reading out an image signal for dynamic range expansionwithout reading out a partial signal for phase difference detection.

In this control, from time t21 to t24, the gain of the column amplifier103 i is set to a high gain by turning the switch 200 on and turning theswitches 201 and 204 off. In this state, the transfer switches 1103 and1104 are turned on at time t22, and the A+B signal is read out. At thistime, the A+B signal read out at a high gain is A/D converted by the A/Dcircuit 104 i during the period from time t22 to t24.

Next, from time t24 to t25, the gain of the column amplifier 103 i isset to a low gain by turning the switch 200 off and turning the switches201 and 204 on. At this time, the A+B signal read out at a low gain isA/D converted by the A/D circuit 104 i during the period from time t24to t25.

FIGS. 6A and 6B are diagrams illustrating the timings at which imagesignals are read out from the image sensor according to the firstembodiment, and illustrate the concept of the signals read out at eachframe as a result of the control illustrated in FIG. 3. In FIGS. 6A and6B, “1H transfer data” refers to one row's worth of data read out fromthe pixel region 100.

FIG. 6A is a diagram illustrating the timing of readout when an imagesignal for phase difference detection, an image signal for phasedifference detection when expanding the dynamic range, and the imagesignal for dynamic range expansion are read out for all rows in a singleframe period. Specifically, the A signal is read out at high gain(called a “high gain A signal” hereinafter), and then the A signal isread out at low gain (called a “low gain A signal” hereinafter).Furthermore, the A+B signal is read out at high gain (called a “highgain A+B signal” hereinafter), and then the A+B signal is read out atlow gain (called a “low gain A+B signal” hereinafter).

Phase difference detection can be carried out by using the high gain Asignal, a high gain B signal obtained by subtracting the high gain Asignal from the high gain A+B signal, the low gain A signal, and a lowgain B signal obtained by subtracting the low gain A signal from the lowgain A+B signal, which have been read out in this manner. The high gainA+B signal and the low gain A+B signal can be used as-is as imagesignals for dynamic range expansion.

Thus with the configuration of the image sensor according to the presentembodiment, the method of readout from the image sensor when reading outone row's worth of data can be changed, and thus an image signal forphase difference detection and an image signal for dynamic rangeexpansion can be read out within the same frame.

However, as illustrated in FIG. 6A, when the image signal for phasedifference detection and the image signal for dynamic range expansionare read out in the same frame, the transfer time of the one frameincreases. Accordingly, when the focus is to be adjusted quickly withoutrecording, such as when the shutter release is pressed halfway beforeshooting a still image or when the screen has been zoomed in foradjusting the focus, a phase difference detection mode is activated.

FIG. 6B is a diagram illustrating the readout timing when only the imagesignal for phase difference detection is read out in the phasedifference detection mode, and illustrates the concept of the signalsread out in each frame as a result of the control illustrated in FIG. 4.As a result of this control, the high gain A signal and the high gainA+B signal are read out. Thus the information for phase differencedetection can be obtained preferentially by increasing the frame rate.

When the focus is determined in advance and focus information is notneeded, such as when shooting a still image, a dynamic range expansionmode is activated. FIG. 7A is a diagram illustrating the readout timingwhen only the image signal for dynamic range expansion is read out inthe dynamic range expansion mode, and illustrates the concept of thesignals read out in each frame as a result of the control illustrated inFIG. 5. As a result of this control, the high gain A+B signal and thelow gain A+B signal are read out. This makes it possible to shorten theshooting time for a single frame.

Meanwhile, if phase difference detection is not needed on thehigh-luminosity side and is only needed near the proper exposure, amethod can also be considered in which the A signal read out at low gainto increase the framerate is not output. FIG. 7B is a diagramillustrating the readout timing in such a case, and here, the high gainA signal, the high gain A+B signal, and the low gain A+B signal are readout. By doing so, the image signal for phase difference detection nearthe proper exposure and the image signal for dynamic range expansion canbe obtained.

In addition to the above, the mode may also be changed in accordancewith the subject. For example, even if the dynamic range expansion modeillustrated in FIG. 7A is active, the mode may be changed to the phasedifference detection mode (FIG. 6B) if the subject to be shot is not ahigh-luminosity subject, and phase difference detection may be carriedout while obtaining the still image.

Furthermore, the order in which the high gain A signal, the low gain Asignal, the high gain A+B signal, and the low gain A+B signal are readout is not limited to the example described above. The order can bechanged as appropriate, e.g., the order in which the low gain signalsand the high gain signals are read out can be reversed, the high gainsignals can be read out first and the low gain signals can be read outthereafter, the low gain signals can be read out first and the high gainsignals can be read out thereafter, and so on.

FIG. 8 is a block diagram illustrating the configuration of an imagecapturing apparatus according to the present embodiment, and illustratesonly constituent elements directly related to the present invention. Theflow of signals when an image signal for phase difference detection andan image signal for dynamic range expansion are output from an imagesensor 400 from the same frame as illustrated in FIG. 6A will bedescribed hereinafter.

The image sensor 400 is the image sensor illustrated in FIGS. 1A and 1B,and the low gain A signal, the high gain A signal, the low gain A+Bsignal, and the high gain A+B signal output from the image sensor 400are input to a distributor 401. The distributor 401 outputs the signalsto be used as an image (the low gain A+B signal and the high gain A+Bsignal) and the signals to be used for phase difference detection (thelow gain A signal, the low gain A+B signal, the high gain A signal, andthe high gain A+B signal) separately.

A B signal generation unit 408 generates the high gain B signal bysubtracting the high gain A signal from the high gain A+B signal outputfrom the distributor 401. The B signal generation unit 408 alsogenerates the low gain B signal by subtracting the low gain A signalfrom the low gain A+B signal.

A phase difference detection unit 403 detects a phase difference fromthe signals for phase difference detection output from the B signalgeneration unit 408 (the high gain A signal, the high gain B signal, thelow gain A signal, and the low gain B signal). A focus computation unit404 carries out focus computation on the basis of the phase differenceinformation detected here and a focus position of a lens. The imagecapturing apparatus notifies a user of focus information, controls thefocus of the lens, and the like on the basis of the obtained focusinformation.

An image synthesizing unit 402 synthesizes an expanded dynamic rangeimage, using any desired synthesizing method, from the signals fordynamic range expansion output from the image sensor, when the high gainA+B signal is saturated. There is, for example, a method of synthesizingwhere a high gain image is used for parts where the subject is dark anda low gain image is used for parts where the subject is bright, but thesynthesizing algorithm is not limited in the present embodiment as longas the method synthesizes two images having different gains. A controlunit 405 controls the switching of shutter speeds, gain, and the like ofthe image sensor 400, changes the readout driving, and so on.

Although the foregoing example describes the flow of signals when thelow gain A signal, the high gain A signal, the low gain A+B signal, andthe high gain A+B signal are read out, the signals may instead be readout as described with reference to FIGS. 6B, 7A, and 7B. In that case,the image signals for phase difference detection and the image signalsfor dynamic range expansion may be output separately to the imagesynthesizing unit 402 and the phase difference detection unit 403 asnecessary by the distributor 401.

FIGS. 9A and 9B show a flowchart illustrating the readout control of theimage sensor 400, carried out by the control unit 405, according to thepresent first embodiment. First, in step S100, the control unit 405determines whether or not it is necessary to expand the dynamic range.The determination as to whether or not it is necessary to expand thedynamic range is carried out when the shutter release is pressed halfwaybefore shooting a still image as described above, based on a shootingmode set by the user, or the like, for example. This determination mayalso be made using the result of determination by the image synthesizingunit 402 whether or not the high gain A+B signal of the image signals inthe previous frame is saturated.

If it is not necessary to expand the dynamic range, the process moves tostep S112, where the high gain A signal for phase difference detectionis read out, after which the process moves to step S113 and the highgain A+B signal is read out. Then, in step S121, it is determinedwhether or not the readout is complete for all of the rows, and if thereadout is not complete, the process returns to step S112 and thereadout is continued. This corresponds to the readout order in the phasedifference detection mode, illustrated in FIG. 6B.

On the other hand, if it is necessary to expand the dynamic range, theprocess moves to step S101, where it is determined whether or not phasedifference detection is necessary. If it is determined that phasedifference detection is not necessary, the process moves to step S110,where the low gain A+B signal is read out, after which the process movesto step S111 and the high gain A+B signal is read out. Then, in stepS122, it is determined whether or not the readout is complete for all ofthe rows, and if the readout is not complete, the process returns tostep S110 and the readout is continued. This corresponds to the readoutorder in the dynamic range expansion mode, illustrated in FIG. 7A.

On the other hand, if phase difference detection is necessary, theprocess moves to step S102, where it is determined whether or not phasedifference detection for high-luminosity parts is necessary. If it isdetermined that phase difference detection for high-luminosity parts isnot necessary, the high gain A signal is read out in step S107, the highgain A+B signal is read out in step S108, and the low gain A+B signal isread out in step S109. Then, in step S123, it is determined whether ornot the readout is complete for all of the rows, and if the readout isnot complete, the process returns to step S107 and the readout iscontinued. This corresponds to the readout order illustrated in FIG. 7B.

If phase difference detection for high-luminosity parts is necessary,the process moves to step S103. Then, the high gain A signal is read outin step S103, the high gain A+B signal is read out in step S104, the lowgain A signal is read out in step S105, and the low gain A+B signal isread out in step S106. Then, in step S124, it is determined whether ornot the readout is complete for all of the rows, and if the readout isnot complete, the process returns to step S103 and the readout iscontinued. This corresponds to the readout order illustrated in FIG. 6A.

The process of FIGS. 9A and 9B ends when one frame's worth of readoutends through any of the above-described readout methods.

According to the present embodiment as described thus far, an imagesignal used for dynamic range expansion and an image signal used forphase difference detection can be obtained in each frame. Additionally,carrying out control so that unnecessary image signals are not read outmakes it possible to increase the frame rate as compared to a case wherethe image signal used for dynamic range expansion and the image signalused for phase difference detection are all read out from every row.

Second Embodiment

A second embodiment of the present invention will be described next.Note that the image sensor according to the second embodiment is thesame as that described in the first embodiment, and thus descriptionsthereof will be omitted here.

FIGS. 10A and 10B are diagrams illustrating the readout timing of imagesignals from the image sensor according to the second embodiment. As inFIGS. 6A and 6B, “1H transfer data” refers to one row's worth of dataread out from the pixel region 100.

FIG. 10A illustrates the readout timing when the image signal for phasedifference detection and the image signal for dynamic range expansionare read out for all lines within a single frame period, withoutcarrying out phase difference detection on the high-luminosity side, andcorresponds to the readout method illustrated in FIG. 7B.

If image signals for phase difference detection and for dynamic rangeexpansion are read out in addition to the normal image signals for alllines in this manner, the data amount will be three times the dataamount resulting from normal readout in which dynamic range expansionand phase difference detection are not carried out, which takes up alarge amount of bandwidth. The framerate will therefore be slower thanwhen reading out only the normal image signal.

Accordingly, in the present embodiment, the image signal for phasedifference detection and the image signal for dynamic range expansionare output in an alternating manner, from row to row, to increase theframerate, as illustrated in FIG. 10B. Thus the frame rate can beincreased by reducing the data amount of the image signals in a singleframe. Furthermore, phase difference detection can be carried out forthe entire area within the screen if the readout method illustrated inFIG. 10B is used, and thus accurate focus control is possible even whenthe user wishes to focus on a desired location.

FIG. 11 is a block diagram illustrating the overall configuration of animage capturing apparatus according to the second embodiment. The imagecapturing apparatus according to the second embodiment adds a pixelinterpolation processing unit 802 to the configuration described in thefirst embodiment with reference to FIG. 8. The rest of the configurationis the same as that illustrated in FIG. 8, and thus the same referencesigns are used, with descriptions omitted as appropriate.

Processing carried out when reading out the image signal for phasedifference detection and the image signal for dynamic range expansion inan alternating manner, from row to row, from the image sensor 400 asillustrated in FIG. 10B will be described next.

The high gain A signal read out at a high gain, the A+B signal read outat a high gain, and the A+B signal read out at a low gain, which havebeen output from the image sensor 400, are input to the distributor 401.The distributor 401 outputs the signals used as an image (the low gainA+B signal and the high gain A+B signal) and the signals used for phasedifference detection (the high gain A signal and the high gain A+Bsignal) separately.

The B signal generation unit 408 generates the high gain B signal bysubtracting the high gain A signal from the high gain A+B signal outputfrom the distributor 401. The phase difference detection unit 403detects a phase difference between the signals for phase differencedetection output from the B signal generation unit 408 (the high gain Asignal and the high gain B signal). The focus computation unit 404carries out focus computation on the basis of the phase differenceinformation detected here and the focus position of a lens. The imagecapturing apparatus notifies a user of focus information, controls thefocus of the lens, and the like on the basis of the obtained focusinformation.

On the other hand, for lines from which the low gain A+B signal is notread out, the pixel interpolation processing unit 802 interpolates thepixel signals from the lines thereabove and therebelow (describedlater). The image synthesizing unit 402 synthesizes an expanded dynamicrange image, using any desired synthesizing method, from the high gainA+B signal and the low gain A+B signal, when the high gain A+B signal issaturated.

Here, FIGS. 12A to 12D illustrate the image data at each of the blocksillustrated in FIG. 11. FIG. 12A illustrates the image signals outputfrom the image sensor 400. As illustrated in FIG. 12A, in the imagesignals output from the image sensor 400, the high gain A signal and thelow gain A+B signal are read out in an alternating manner between thehigh gain A+B signals.

FIG. 12B illustrates the image signals separated by the distributor 401and input to the pixel interpolation processing unit 802. The high gainA signal used for phase difference detection is not necessary forexpanding the dynamic range and is therefore thinned out by thedistributor 401, and thus the lines in which the high gain A signal forphase difference detection is read out lack the low gain A+B signal fordynamic range expansion.

FIG. 12C is a diagram illustrating the image signals output from thepixel interpolation processing unit 802. In the lines where the highgain A signal is read out, the low gain A+B signal is interpolated usingthe low gain A+B signals that are adjacent above and below.

Finally, FIG. 12D is a diagram illustrating the image signals input tothe B signal generation unit 408. Because the signals for dynamic rangeexpansion are not needed in the phase difference detection process, thelow gain A+B signal and the high gain A+B signal, which are imagesignals for dynamic range expansion in the lines where the high gain Asignal is not read out, are thinned out by the distributor 401.

According to the present second embodiment as described thus far,dynamic range expansion and phase difference detection can be carriedout throughout the entire pixel region even when fewer image signals areread out.

Third Embodiment

A third embodiment of the present invention will be described next. Notethat the image sensor according to the third embodiment is also the sameas that described in the first embodiment, and thus descriptions thereofwill be omitted here.

In the above-described second embodiment, an image signal for dynamicrange expansion is generated through interpolation using the low gainA+B signals adjacent above and below for lines in which the high gain Asignal for phase difference detection is read out. However,interpolating the image signal from above and below reduces the verticalresolution. Accordingly, in the present embodiment, the luminance levelof the image signal for phase difference detection is detected, and whenthe luminance level is less than or equal to a prescribed value and thephase difference is less than or equal to a prescribed value, the imagesignal for phase difference detection, rather than the interpolated lowgain A+B signal, is used as the image signal for dynamic rangeexpansion, which prevents a drop in the vertical resolution.

FIG. 13 is a block diagram illustrating the overall configuration of animage capturing apparatus according to the third embodiment. The imagecapturing apparatus according to the present embodiment adds a luminancedetection unit 902 and a line selection processing unit 903 to theconfiguration described in the second embodiment with reference to FIG.11. The processing by the distributor 401 also differs from thatillustrated in FIG. 11. The rest of the configuration is the same asthat described in the first embodiment with reference to FIG. 8 and inthe second embodiment with reference to FIG. 11, and thus the samereference signs are used, with descriptions omitted as appropriate.

In the third embodiment, the distributor 401 outputs three signals,namely the high gain A signal, the low gain A+B signal, and the highgain A+B signal, as signals used for images. The signals output from thedistributor 401 are input to the luminance detection unit 902. Theluminance detection unit 902 detects the luminance of the high gain Asignal and outputs the detection result to the line selection processingunit 903. The line selection processing unit 903 determines whether ornot to use the high gain A signal as the image signal for dynamic rangeexpansion on the basis of the information from the luminance detectionunit 902 and the phase difference detection unit 403.

FIG. 14 is a flowchart illustrating processing performed in the thirdembodiment. First, in step S300, the luminance detection unit 902detects the luminance level of the high gain A signal of the input line.Then, in step S301, the phase difference detection unit 403 detects aphase difference between the high gain A signal and the high gain Bsignal.

In step S302, the line selection processing unit 903 determines whetheror not the luminance level of the high gain A signal detected in stepS300 is less than or equal to a prescribed value Th1 (less than or equalto a threshold). The process moves to step S303 if the level is lessthan or equal to the prescribed value Th1, and to step S305 if the levelis greater than the prescribed value Th1.

In step S303, it is determined whether or not the phase differencebetween the high gain A signal and the high gain B signal is less thanor equal to a prescribed value Th2 on the basis of the detection resultfrom step S301. If the difference is less than or equal to theprescribed value Th2, it is determined that the focus state is near anin-focus state, and the process moves to step S304, whereas if thedifference is greater than the prescribed value Th2, the process movesto step S305.

In step S304, the high gain A signal is selected as the image signal fordynamic range expansion. On the other hand, in step S305, the low gainA+B signal interpolated from above and below is selected to be used asthe image signal for dynamic range expansion, and the pixelinterpolation processing unit 802 generates data interpolated from aboveand below in step S306. In step S307, the image synthesizing unit 402carries out a process for expanding the dynamic range using the highgain A signal or the low gain A+B signal.

According to the third embodiment as described above, the dynamic rangecan be expanded without reducing the resolution in the verticaldirection by using the high gain A signal when the luminance level ofthe high gain A signal is less than or equal to a predeterminedluminance and the focus state is in focus or near an in-focus state.

Fourth Embodiment

A fourth embodiment of the present invention will be described next.Note that the image sensor according to the fourth embodiment is alsothe same as that described in the first embodiment, and thusdescriptions thereof will be omitted here.

The fourth embodiment adds, to the third embodiment, a process fordetecting a movement amount in an object when shooting a moving image,changing the driving of the image sensor if the movement amount is lessthan or equal to a prescribed movement amount, and switching betweenreadout for phase difference detection and readout for dynamic rangeexpansion from frame to frame. A drop in the vertical resolution isprevented by using the image signals from the previous and next framesduring image synthesizing.

FIG. 15 is a block diagram illustrating the overall configuration of animage capturing apparatus according to the fourth embodiment. Theconfiguration illustrated in FIG. 15 adds a motion vector detection unit909 and memory 910 to the configuration described in the thirdembodiment with reference to FIG. 13. The rest of the configuration isthe same as that illustrated in FIG. 13, and thus the same referencesigns are used, with descriptions omitted as appropriate.

The motion vector detection unit 909 detects the movement amount of anobject and outputs the detection result to the image synthesizing unit402 and the control unit 405. The memory 910 can temporarily hold imagesignals, and an image can be synthesized using the image signals fromthe previous and next frames.

FIG. 16 is a flowchart illustrating processing performed in the fourthembodiment. Processes that are the same as the processes described inthe third embodiment with reference to FIG. 14 are given the same stepnumbers, with descriptions omitted as appropriate.

When, in step S304, the high gain A signal is selected as the imagesignal for dynamic range expansion, in step S407, the image synthesizingunit 402 carries out a process for expanding the dynamic range by usingthe high gain A signal, as described in the third embodiment.

On the other hand, if the high gain A signal is not selected, theprocess moves from step S303 to step S408, and in step S408, it isdetermined whether or not the movement amount of the object is greaterthan or equal to a prescribed value Th3. If the amount is greater thanor equal to the prescribed value Th3, the process moves to step S305,where the data, interpolated from the low gain A+B signals in the pixelsabove and below, is selected. In step S306, the pixel interpolationprocessing unit 802 generates the interpolated data, and in step S410,the image synthesizing unit 402 carries out a process for expanding thedynamic range using the low gain A+B signal.

On the other hand, if the movement amount of the object is less than theprescribed value Th3, in step S409, the driving of the image sensor 400is changed. Here, a process for alternating between reading out the highgain A signal and the high gain A+B signal (the readout methodillustrated in FIG. 6B) and reading out the high gain A+B signal and thelow gain A+B signal (the readout method illustrated in FIG. 7A) fromframe to frame is carried out. Then, in step S411, a process forexpanding the dynamic range using the signal interpolated from the lowgain A+B signal of the previous and next frames, held in the memory 910,is carried out.

According to the fourth embodiment as described thus far, a process forexpanding the dynamic range using a signal interpolated from the lowgain A+B signals in the previous and next frames is carried out whenthere is little movement in an object, which makes it possible tosuppress a drop in the vertical resolution.

Fifth Embodiment

A fifth embodiment of the present invention will be described next. Notethat the image sensor according to the fifth embodiment is the same asthat described in the first embodiment, and thus descriptions thereofwill be omitted here.

FIG. 17 is a diagram illustrating details from the unit pixel 101 to theA/D circuit group 104 in the image sensor, in which vertical outputlines, column amplifiers, and the like are connected to both thephotoelectric conversion portions A and B. In this configuration, thesignals from the photoelectric conversion element 1101 corresponding tothe photoelectric conversion portion A and the photoelectric conversionelement 1102 corresponding to the photoelectric conversion portion B,illustrated in FIG. 1B, can be read out in parallel to the verticaloutput line 1113 and a vertical output line 1115, respectively.

In FIG. 17, elements that are the same as those illustrated in FIG. 2Aare given the same reference signs, and descriptions thereof areomitted. The configuration illustrated in FIG. 17 adds, to theconfiguration illustrated in FIG. 2A, a charge holding unit 1108 and apixel amplifier 1114 for reading out the partial signals from thephotoelectric conversion element 1102 independently, and a currentcontrol unit 1116 for controlling current in the vertical output line1115, in each of the pixels 101.

The column amplifier 103 i includes two amplifiers for amplifying thesignals output from the photoelectric conversion element 1101 and thephotoelectric conversion element 1102 to the vertical output lines 1113and 1115, respectively, and outputs the amplified signals to the A/Dcircuit group 104 in a later stage. Additionally, in addition to theconfiguration illustrated in FIG. 2A, each A/D circuit 104 i includes anA/D conversion unit 1118 for converting an analog signal from thephotoelectric conversion element 1102 into a digital signal, and memory1119 and memory 1120 for temporarily holding the digital signal.

According to this configuration, the partial signals from thephotoelectric conversion elements 1101 and 1102 can be read out,processed, and output in parallel, which, although increasing thecircuit scale, also makes it possible to shorten the readout time.

Additionally, the A+B signal can be obtained by adding the read out Asignal and the B signal in the signal processing unit 106. However, inthis case, the B signal generation unit 408 illustrated in FIGS. 8, 11,and 13 is not necessary, whereas an A+B signal generation unit isnecessary between the distributor 401 and the image synthesizing unit402.

FIG. 18 is a timing chart illustrating control of the column amplifier103 i, carried out when high gain and low gain partial signals are readout in parallel from the photoelectric conversion elements 1101 and 1102in the configuration illustrated in FIG. 17.

First, from time t51 to t54, the gain of the column amplifier 103 i isset to a high gain by turning the switch 200 on and turning the switches201 and 204 off. In this state, the transfer switches 1103 and 1104 areturned on at time t52, and the A signal and B signal are read out. Atthis time, the A signal and the B signal read out at a high gain are A/Dconverted by the A/D circuit 104 i during the period from time t52 tot54.

Next, from time t54 to t55, the gain of the column amplifier 103 i isset to a low gain by turning the switch 200 off and turning the switches201 and 204 on. At this time, the A signal and B signal read out at alow gain are A/D converted by the A/D circuit 104 i during the periodfrom time t54 to t55.

Among the read-out high gain A signal, high gain B signal, low gain Asignal, and low gain B signal, the high gain A+B signal is generated bythe signal processing unit 106 illustrated in FIG. 1A by adding the highgain A signal and the high gain B signal, and is then output.Additionally, the low gain A+B signal is generated by the signalprocessing unit 106 by adding the low gain A signal and the low gain Bsignal, and is then output.

Although the timing chart illustrated in FIG. 18 indicates a case whereboth high gain and low gain signals are read out, the present inventionis not limited thereto. For example, the readout speed can be increasedby reading out only the signals with the necessary gain, such as whenthe high gain signal is not necessary, the low gain signal is notnecessary, or the like.

According to the present fifth embodiment as described thus far, animage signal used for dynamic range expansion and an image signal usedfor phase difference detection can be obtained in each frame, withoutreducing the framerate.

If the low gain A+B signal is to be interpolated using the image signalsfrom the pixels above and below, the interpolation ratio of the pixelsabove and below may be changed as desired in accordance with thecircumstances.

Sixth Embodiment

A sixth embodiment of the present invention will be described next. FIG.19 is a block diagram illustrating an image capturing system 10according to the sixth embodiment, which is constituted primarily by amain body 11 and an interchangeable lens 12 that can be removed from themain body 11.

The interchangeable lens 12 is a shooting lens constituted by a group ofa plurality of lenses, and includes a focus lens, a zoom lens, and ashift lens in addition to an aperture. The focal length of theinterchangeable lens 12 can be changed by electrical signals from afocus control unit 19 (described later).

The main body 11 includes an image sensor 13, an image processing unit14, an HDR synthesizing unit 15, a phase difference detection unit 16,an image output unit 17, a CPU 18, the focus control unit 19, and a bus20. The connection parts for the interchangeable lens 12, the imageoutput unit 17, and the like are exposed on the surface of the main body11.

The image sensor 13 is configured as illustrated in FIGS. 1, 2A, and 2B,described above.

The image processing unit 14 corrects level differences produced by theimage sensor 13. For example, the level of pixels in an active regionare corrected using pixels in an optical black (OB) region, anddefective pixels are corrected using neighbor pixels. The imageprocessing unit 14 also performs various types of processes such ascorrection for decreases in ambient light amounts, color correction,edge enhancement, noise reduction, gamma correction, gain, and so on.The image processing unit 14 carries out these processes on RAW imagedata output from the image sensor 13, and outputs the corrected imagedata to various other units.

The HDR synthesizing unit 15 carries out HDR synthesizing (describedlater) using the high gain A signal, low gain A signal, high gain A+Bsignal, and low gain A+B signal obtained by driving the image sensor 13as illustrated in FIG. 3. An HDR-A signal and an HDR-(A+B) signal havingexpanded dynamic ranges are generated as a result. The processing by theHDR synthesizing unit 15 according to the present embodiment will bedescribed in detail later with reference to FIGS. 20A and 20B.

The phase difference detection unit 16 obtains an HDR-B signal for eachunit pixel 101, from the HDR-A signal and the HDR-(A+B) signal obtainedfrom the HDR synthesizing unit 15. The HDR-A signals from the unitpixels 101 are collected to generate an A image signal, and the HDR-Bsignals from the unit pixels 101 are collected to generate a B imagesignal. A correlation between the A image signal and the B image signalis then computed to calculate information such as a defocus amount,various types of reliabilities, and so on. A defocus amount on the imageplane is calculated as the defocus amount, on the basis of shift betweenthe A image signal and the B image signal. The defocus amount has apositive or negative value, and whether the focus is in front of thesubject or behind the subject can be determined by whether the defocusamount has a positive value or a negative value. The extent to which thesubject is out of focus can be determined from the absolute value of thedefocus amount, and the subject is determined to be in focus when theabsolute value of the defocus amount is within a prescribed value near0. The phase difference detection unit 16 outputs information indicatingwhether the focus is in front of the subject or behind the subject tothe CPU 18 on the basis of the calculated defocus amount. Additionally,information indicating the degree of focus, corresponding to the degreeto which the subject is out of focus, is output to the CPU 18 and thelike on the basis of the absolute value of the defocus amount. Theinformation as to whether the focus is in front of the subject or behindthe subject is output when the defocus amount is greater than aprescribed value, and information indicating that the subject is infocus is output when the absolute value of the defocus amount is withinthe prescribed value.

The CPU 18 adjusts the focus by controlling the interchangeable lens 12through the focus control unit 19 in accordance with the defocus amountfound by the phase difference detection unit 16.

The image output unit 17 outputs the image signal synthesized by the HDRsynthesizing unit 15 to the exterior of the image capturing system 10.In addition to adjusting the focus as mentioned here, the CPU 18controls the various units in the image capturing system 10 inaccordance with control software.

The constituent elements described thus far are connected to theinternal bus 20, which is a transfer path for control signals, datasignals, and the like among the constituent elements.

FIGS. 20A and 20B are graphs illustrating the concept of an HDRsynthesizing process based on the input luminosities of image signalsinput to the HDR synthesizing unit 15 (the high gain A+B signal and thelow gain A+B signal). FIG. 20A is a graph illustrating a case whereoffset of the high gain A+B signal is higher than offset of the low gainA+B signal, whereas FIG. 20B is a graph illustrating a case where theoffset of the high gain A+B signal is lower than the offset of the lowgain A+B signal.

In the present embodiment, the HDR synthesizing unit 15 carries outoperations in accordance with the luminosities of the high gain A+Bsignal and the low gain A+B signal input for each pixel. The high gainA+B signal and the low gain A+B signal are obtained by applyingdifferent gains to the same A+B signal, and thus the luminositiescorresponding to the high gain A+B signal and the low gain A+B signalare the same. Accordingly, the high gain A+B signal and the low gain A+Bsignal are subjected to HDR synthesis by applying different weightsdepending on the luminosity.

When the luminosity of the A+B signal is within a predeterminedsynthesizing range as indicated in FIGS. 20A and 20B, the input highgain A+B signal is multiplied by a weighting coefficient (1−α), and thelow gain A+B signal by a coefficient α, after which the signals areadded and output. At this time, the value of α is varied in a linearmanner between 0 and 1 in accordance with the luminosity within thesynthesizing range, with α being 0 at minimum luminosity and a being 1at maximum luminosity in the synthesizing range. On the other hand, ifthe luminosity is lower than the synthesizing range, the high gain A+Bsignal is multiplied by the coefficient (1−α), and the low gain A+Bsignal by the coefficient α, assuming α=0, after which the signals areadded and output. In other words, the high gain A+B signal is output. Ifthe luminosity is higher than the synthesizing range, the high gain A+Bsignal is multiplied by the coefficient (1−α), and the low gain A+Bsignal by the coefficient α, assuming α=1, after which the signals areadded and output. In other words, the low gain A+B signal is output.

Here, it is also conceivable to carry out the HDR synthesizing on thehigh gain A signal and the low gain A signal according tocharacteristics such as those indicated in FIGS. 20A and 20B. However,if there is a difference between the offsets of the low gain A+B signaland the high gain A+B signal as indicated in FIGS. 20A and 20B, aproblem arises in that the linearity of the input luminosity and theoutput luminosity cannot be ensured between the synthesizing range andparts not in the synthesizing range. This problem will be describednext, using a case where the offset of the high gain signal is higherthan the offset of the low gain signal as an example.

For example, when the luminosities corresponding to the A signal and the(A+B) signal are within the synthesizing range for a given unit pixel asindicated in FIG. 21, if the luminosities of the input signals arerepresented by Ain and (A+B)in, respectively, the output signals are theHDR-A signal and the HDR-(A+B) signal. Here, assuming a luminositydifference ΔBin (=(A+B)in −Ain) for the input signal corresponding tothe B signal, the post-synthesis HDR-B signal is obtained from(HDR-(A+B) signal)−(HDR-A signal). This will be called “ΔB1out” for thesake of simplicity.

Meanwhile, assume that luminosities A′in and (A+B)′in, whichrespectively correspond to the A signal and the (A+B) signal for a givenpixel, are higher than the synthesizing range, and the luminositydifference of an input signal corresponding to the B signal is the sameas the above-described luminosity difference ΔBin. In this case, thepost-HDR synthesizing HDR-B signal is (HDR-(A+B)′signal)−(HDR-A′signal).This will be called “ΔB2out” for the sake of simplicity. In this manner,although HDR-B signals for the same luminosity difference ΔBin shouldhave the same value, the signals ultimately have different values, asindicated by ΔB2out and ΔB1out.

If the luminosities of the A signal and the (A+B) signal in each pixelare both outside the synthesizing range, a post-HDR synthesis signal isgenerated by applying the same weighting to the A signal and the (A+B)signal. However, if even one of these is within the synthesizing range,different weights are applied to the A signal and the (A+B) signal,resulting in different values. Thus if different HDR-B signals have beenobtained for the same B signal, the correlation amount cannot bedetected correctly, and thus the correct defocus amount cannot beobtained.

Accordingly, in the present embodiment, if the luminosity correspondingto the A signal is lower than the luminosity corresponding to the A+Bsignal in the same unit pixel but the A signal is subjected to HDRsynthesizing, the HDR synthesizing is carried out using the samecoefficient α as that used when subjecting the A+B signal output fromthe same unit pixel to the HDR synthesizing. FIG. 22 is a diagramillustrating this HDR synthesizing process.

As illustrated in FIG. 22, when subjecting the high gain A signal andthe low gain A signal to HDR synthesizing, using the same coefficient αas that used when subjecting the high gain A+B signal and the low gainA+B signal from the same unit pixel to HDR synthesizing results in theslope within the synthesizing range being the same as the slope outsideof the synthesizing range. As such, the same ΔB2out and ΔB1out can beobtained for the same B signal, i.e., the same HDR-B signal can beobtained.

Next, operations from the readout of signals from the image sensor 13 tothe on-image plane phase difference AF according to the sixth embodimentwill be described using FIG. 23.

First, in step S600, the image sensor 13 photoelectrically converts asubject optical image formed by the interchangeable lens 12 at eachpixel, and outputs the A signal and the A+B signal as described above.Different gains are applied to the A signal and the A+B signal by thecolumn amplifier group 103, after which the signals undergoanalog-digital conversion by the A/D circuit group 104, and the highgain A signal, the low gain A signal, the high gain A+B signal, and thelow gain A+B signal are output as a result.

Next, in step S601, the image processing unit 14 corrects the level ofpixels in the active region, corrects defective pixels using pixelslocated in the periphery thereof, and performs correction for decreasesin ambient light amounts, color correction, edge enhancement, noiseremoval, and gamma correction on the high gain A+B signal and the lowgain A+B signal. The low gain A+B signal and the low gain A signal arethen subjected to gain correction to provide the same gain for the highgain A+B signal and the high gain A signal, after which the processmoves to step S602.

In step S602, the HDR synthesizing unit 15 finds the coefficient α to beused in the HDR synthesizing, for each of the unit pixels, in accordancewith the luminosity of the low gain A+B signal processed by the imageprocessing unit 14, after which the process moves to step S603.

In step S603, the HDR synthesizing unit 15 multiplies the low gain A+Bsignal by the coefficient α, multiplies the high gain A+B signal by thecoefficient (1−α), and adds the signals to synthesize the signals.Likewise, using the value of the coefficient α used in the HDR synthesisof the low gain A+B signal and the high gain A+B signal from the samepixel, the low gain A signal is multiplied by the coefficient α, and thehigh gain A signal by the coefficient (1−α), after which the signals areadded to synthesize the signals. The process then moves to step S604.

In step S604, the phase difference detection unit 16 calculates theHDR-B signal by subtracting the HDR-A signal from the synthesizedHDR-(A+B) signal, and the process then moves to step S605.

In step S605, the phase difference detection unit 16 calculates theon-image plane defocus amount by finding the phase difference betweenthe HDR-A signal and the HDR-B signal, after which the process moves tostep S606.

In step S606, the CPU 18 determines whether or not the focus state is anin-focus state from the calculated defocus amount, and if the focusstate is an in-focus state, the process ends, where as if the focusstate is not an in-focus state, the process moves to step S607.

In step S607, the focus control unit 19 changes the focal length of theinterchangeable lens 12 in accordance with the defocus amount, afterwhich the process returns to step S600, and the processing describedthus far is repeated.

According to the present sixth embodiment as described thus far, the lowgain A signal and the high gain A signal, which are signals for phasedifference detection, are subjected to HDR synthesizing using the samecoefficient α as that used for the low gain A+B signal and the high gainA+B signal for image capturing, from the same pixel. This makes itpossible to obtain the correct B signal without being affected by ashift between the low gain and high gain offsets. The defocus amount cantherefore be calculated correctly, which makes it possible to carry outthe on-imaging plane phase difference AF appropriately.

Seventh Embodiment

A seventh embodiment of the present invention will be described next.The configuration of an image capturing system according to the seventhembodiment is the same as that illustrated in FIG. 19, but the methodfor reading out signals from the image sensor 13, and the processing bythe HDR synthesizing unit 15 and the phase difference detection unit 16,are different from the sixth embodiment. As such, the differences willbe described below.

In the above-described sixth embodiment, the A+B signal and the A signalare output from each unit pixel in the image sensor 13. However, in theseventh embodiment, one A signal is output from the image sensor 13 forevery eight pixels' worth of the A+B signal for image capturing.

FIGS. 24A and 24B are diagrams illustrating the concept of reading outthe A+B signal and the A signal output from the image sensor 13. FIG.24A illustrates the concept of the signals output from the image sensor13 according to the sixth embodiment, where the A+B signal and the Asignal are output from each unit pixel. By contrast, FIG. 24Billustrates the concept of the signals output from the image sensor 13according to the seventh embodiment, where one A signal is output forevery eight pixels' worth of the A+B signal. In the present seventhembodiment, the A signal is assumed to have a value obtained by takingthe arithmetic mean of the eight pixels' worth of A signalscorresponding to the eight pixels' worth of A+B signals. To distinguishthis from the A signals output from each unit pixel, the A signalaccording to the seventh embodiment will be called an “A′ signal”hereinafter.

The present invention is not limited to the arithmetic mean in thismanner, and thinning readout may be used; additionally, the number ofpixels is not limited to eight. For example, if one A′ signal is to beoutput for 3×3, i.e., 9 pixels' worth of A+B signals, it is conceivableto use the A signal in the unit pixel in the center as the A′ signalaccording to the seventh embodiment. Additionally, the number of pixelscorresponding to a single A′ signal may be changed in accordance withthe frame rate of the image output from the image sensor 13, the totalnumber of pixels output in a single frame period, and the like.

The HDR synthesizing unit 15 calculates the coefficient α, which is usedto synthesize the A+B signal, and a coefficient α′, which is used tosynthesize the A′ signal. The coefficient α′ is the average value ofeight coefficients α calculated to subject the eight pixels' worth ofhigh gain A+B signals and low gain A+B signals to HDR synthesizing, anda high gain A′ signal and a low gain A′ signal corresponding to thoseeight pixels are subjected to HDR synthesizing using the coefficient α′.

Next, operations from the readout of signals from the image sensor 13 tothe on-image plane phase difference AF according to the seventhembodiment will be described using FIG. 25. Note that in FIG. 25,processes that are the same as those in the flowchart of FIG. 23 aregiven the same reference signs, and descriptions thereof are omitted asappropriate.

First, in step S700, a subject optical image formed by theinterchangeable lens 12 is photoelectrically converted at the pixellevel, and the A signals and the A+B signals are output as describedabove. The A′ signal, which is the average value of the eight pixels'worth of A signals, is then found. Different gains are applied by thecolumn amplifier group 103 to the A′ signal and the A+B signal that havebeen found, after which the signals undergo analog-digital conversion bythe A/D circuit group 104, and the high gain A+B signal and low gain A+Bsignal for each pixel, as well as the high gain A′ signal and the lowgain A′ signal, are output. Next, in step S601, the processing describedabove with reference to FIG. 23 is carried out.

In step S702, the HDR synthesizing unit 15 calculates the coefficient αto be used in the HDR synthesizing, on a pixel-by-pixel basis, inaccordance with the luminosities of the A+B signals from the eightpixels processed by the image processing unit 14. Furthermore, thecoefficient α′, which is the average value of the eight coefficients αfor the eight pixels, is calculated, after which the process moves tostep S703.

In step S703, the HDR synthesizing unit 15 multiplies the low gain A+Bsignal from each unit pixel by the coefficient α found in step S702,multiplies the high gain A+B signal by the coefficient (1−α), and addsthe signals to synthesize the signals. On the other hand, using thecoefficient α′ obtained from the coefficients α for the eightcorresponding pixels, the low gain A′ signal is multiplied by thecoefficient α′, the high gain A′ signal is multiplied by a coefficient(1−α′), and the signals are added to synthesize the signals. The processthen moves to step S704.

In step S704, the phase difference detection unit 16 calculates eightpixels' worth of the HDR-B signals by subtracting the synthesized HDR-A′signal from each of the HDR-(A+B) signals from the corresponding eightpixels, after which the process moves to step S605. The processing thatfollows thereafter is the same as that described in the sixthembodiment.

According to the present seventh embodiment as described thus far, thelow gain A′ signal and the high gain A′ signal, which are signals forphase difference detection, are subjected to HDR synthesizing using thecoefficient α′, which is the average value of the coefficients α usedwhen synthesizing the low gain A+B signals and the high gain A+B signalsfrom a plurality of corresponding unit pixels. This makes it possible toobtain the correct B signal without being affected by a shift betweenthe low gain and high gain offsets, even when a single A′ signal isoutput for a plurality of A+B signals. The defocus amount can thereforebe calculated correctly, which makes it possible to carry out theon-imaging plane phase difference AF appropriately.

Furthermore, although the present embodiment describes taking the simpleaverage value of the coefficients α for the corresponding pixels as thecoefficient α′, an average value weighted in accordance with theluminosity sensitivity of each color may be used instead.

Additionally, although the above example describes using the A imagesignal obtained from the A signals and the B image signal obtained fromthe B signals for focus detection, the A image signal and the B imagesignal are not limited to being used in this manner. The signals can beused for other applications, such as generating a pair of signals havingparallax for a three-dimensional display.

Although the foregoing has described preferred embodiments of thepresent invention, the present invention is not intended to be limitedto the specific embodiments, and all variations that do not depart fromthe essential spirit of the invention are intended to be included in thescope of the present invention. Some of the above-described embodimentsmay be combined as appropriate.

Other Embodiments

The present invention may be applied in a system constituted by aplurality of devices (e.g., a host computer, an interface device, ascanner, a video camera, and the like) or in an apparatus constituted bya single device.

Embodiment(s) of the present invention can also be realized by acomputer of a system or apparatus that reads out and executes computerexecutable instructions (e.g., one or more programs) recorded on astorage medium (which may also be referred to more fully as a‘non-transitory computer-readable storage medium’) to perform thefunctions of one or more of the above-described embodiment(s) and/orthat includes one or more circuits (e.g., application specificintegrated circuit (ASIC)) for performing the functions of one or moreof the above-described embodiment(s), and by a method performed by thecomputer of the system or apparatus by, for example, reading out andexecuting the computer executable instructions from the storage mediumto perform the functions of one or more of the above-describedembodiment(s) and/or controlling the one or more circuits to perform thefunctions of one or more of the above-described embodiment(s). Thecomputer may comprise one or more processors (e.g., central processingunit (CPU), micro processing unit (MPU)) and may include a network ofseparate computers or separate processors to read out and execute thecomputer executable instructions. The computer executable instructionsmay be provided to the computer, for example, from a network or thestorage medium. The storage medium may include, for example, one or moreof a hard disk, a random-access memory (RAM), a read only memory (ROM),a storage of distributed computing systems, an optical disk (such as acompact disc (CD), digital versatile disc (DVD), or Blu-ray Disc (BD)™),a flash memory device, a memory card, and the like.

While the present invention has been described with reference toexemplary embodiments, it is to be understood that the invention is notlimited to the disclosed exemplary embodiments. The scope of thefollowing claims is to be accorded the broadest interpretation so as toencompass all such modifications and equivalent structures andfunctions.

This application claims the benefit of Japanese Patent Applications No.2018-078625, filed on Apr. 16, 2018, No. 2019-021823, filed on Feb. 8,2019, and No. 2019-021825, filed on Feb. 8, 2019 which are herebyincorporated by reference herein in their entirety.

What is claimed is:
 1. An image sensor comprising: a pixel regionincluding a plurality of microlenses arranged in a matrix, and aplurality of photoelectric conversion portions provided for each of themicrolenses; a plurality of amplifiers that apply a plurality ofdifferent gains to signals output from the pixel region; and a scanningcircuit that scans the pixel region so that a partial signal and anadded signal are read out, the partial signal being a signal from someof the plurality of photoelectric conversion portions, and the addedsignal being a signal obtained by adding the signals from the pluralityof photoelectric conversion portions.
 2. An image capturing apparatuscomprising: an image sensor, the image sensor including a pixel regionhaving a plurality of microlenses arranged in a matrix and a pluralityof photoelectric conversion portions provided for each of themicrolenses, a plurality of amplifiers that can amplify signals outputfrom the pixel region using a plurality of different gains including atleast a first gain, and a scanning circuit that scans the pixel regionso that a partial signal and an added signal are read out, the partialsignal being a signal from some of the plurality of photoelectricconversion portions, and the added signal being a signal obtained byadding the signals from the plurality of photoelectric conversionportions; a controller that controls the scanning circuit; a processorthat expands a dynamic range using the added signal amplified using theplurality of different gains; and a focus detection circuit that carriesout phase difference focus detection using the partial signal and theadded signal both of which are amplified using the first gain.
 3. Theimage capturing apparatus according to claim 2, wherein in the casewhere the image capturing apparatus is in a predetermined first mode,the controller: controls the scanning circuit to read out the addedsignal and the partial signal from the pixel region; and controls theplurality of amplifiers to amplify the added signal and the partialsignal using the plurality of different gains.
 4. The image capturingapparatus according to claim 2, wherein in the case where the imagecapturing apparatus is in a predetermined second mode, the controller:controls the scanning circuit to read out the added signal and thepartial signal from the pixel region; and controls the plurality ofamplifiers to amplify the added signal and the partial signal using thefirst gain.
 5. The image capturing apparatus according to claim 2,wherein in the case where the image capturing apparatus is in apredetermined third mode, the controller: controls the scanning circuitto read out the added signal from the pixel region; and controls theplurality of amplifiers to amplify the added signal using the pluralityof different gains.
 6. The image capturing apparatus according to claim2, wherein in the case where the image capturing apparatus is in apredetermined fourth mode, the controller: controls the scanning circuitto read out the added signal and the partial signal from the pixelregion; and controls the plurality of amplifiers to amplify the partialsignal using the first gain and to amplify the added signal using theplurality of different gains.
 7. The image capturing apparatus accordingto claim 2, wherein in the case where the image capturing apparatus isin a predetermined fifth mode, the controller: controls the scanningcircuit to read out a first row, in which the added signal and thepartial signal are read out, and a second row, in which the added signalis read out, from the pixel region in an alternating manner; andcontrols the plurality of amplifiers to amplify the added signal and thepartial signal read out from the first row using the first gain, and toamplify the added signal read out from the second row using theplurality of different gains.
 8. The image capturing apparatus accordingto claim 7, further comprising: an interpolation circuit thatinterpolates each of the first rows using the added signals read outfrom adjacent ones of the second rows and amplified using gains asidefrom the first gain, wherein the processor expands the dynamic range ofthe first row using the added signal read out from the first row andamplified using the first gain, and the signal interpolated by theinterpolation circuit.
 9. The image capturing apparatus according toclaim 8, wherein in the case where a luminance level of the partialsignal read out from the first row and amplified using the first gain isless than or equal to a predetermined threshold and the focus statedetected by the focus detection circuit is closer to an in-focus statethan a predetermined focus state, the processor expands the dynamicrange of the first row using the added signal and the partial signalread out from the first row and amplified using the first gain.
 10. Theimage capturing apparatus according to claim 8, further comprising: amotion detector that detects motion in an object, wherein in the casewhere the motion of the object is less than or equal to a predeterminedthreshold, the controller: carries out control so that a first frame anda second frame alternate, the first frame being a frame in which thescanning circuit is controlled to read out the added signal and thepartial signal from the pixel region and the plurality of amplifiers iscontrolled to amplify the added signal and the partial signal using thefirst gain, and the second frame being a frame in which the scanningcircuit is controlled to read out the added signal from the pixel regionand the plurality of amplifiers is controlled to amplify the addedsignal using the plurality of different gains; the interpolation circuitinterpolates the first frame using the added signal amplified usinggains aside from the first gain in the second frame adjacent to thefirst frame; and the processor expands the dynamic range of the firstframe using the added signal of the first frame amplified using thefirst gain, and the signal interpolated by the interpolation circuit.11. An image sensor comprising: a pixel region including a plurality ofmicrolenses arranged in a matrix, and a plurality of photoelectricconversion portions provided for each of the microlenses; a plurality ofamplifiers that apply a plurality of different gains to signals outputfrom the pixel region; and a scanning circuit that scans the pixelregion so that signals are read out in parallel from each of theplurality of photoelectric conversion portions.
 12. An image capturingapparatus comprising: an image sensor, the image sensor including apixel region having a plurality of microlenses arranged in a matrix anda plurality of photoelectric conversion portions provided for each ofthe microlenses, a plurality of amplifiers that amplify signals outputfrom the pixel region using a plurality of different gains including atleast a first gain, and a scanning circuit that scans the pixel regionso that signals are read out in parallel from each of the plurality ofphotoelectric conversion portions; a controller that controls thescanning circuit; a processor that expands a dynamic range using anadded signal, the added signal obtained by adding signals amplifiedusing the plurality of different gains; and a focus detection circuitthat carries out phase difference-type focus detection using signalsoutput from each of the plurality of photoelectric conversion portions,amplified using the first gain.
 13. The image capturing apparatusaccording to claim 12, wherein in the case where the image capturingapparatus is in a predetermined first mode, the controller controls theplurality of amplifiers to amplify the signals output from each of theplurality of photoelectric conversion portions using the plurality ofdifferent gains.
 14. The image capturing apparatus according to claim12, wherein in the case where the image capturing apparatus is in apredetermined second mode, the controller controls the plurality ofamplifiers to amplify the signals output from each of the plurality ofphotoelectric conversion portions using the first gain.
 15. An imageprocessing apparatus comprising: a synthesizer that carries out dynamicrange expansion synthesis, using a first coefficient, on a plurality ofimage signals amplified at a plurality of different gains and outputfrom an image sensor, and carries out dynamic range expansion synthesis,using a second coefficient, on a plurality of first signals amplified atthe plurality of different gains using pupil-divided partial signals andoutput from the image sensor; and a processor that finds a second signalby subtracting the synthesized first signals from the synthesized imagesignals corresponding to the synthesized first signals, wherein thesynthesizer determines the second coefficient on the basis of the firstcoefficient.
 16. The image processing apparatus according to claim 15,wherein the synthesizer determines the first coefficient in accordancewith luminosities corresponding to the plurality of image signals. 17.The image processing apparatus according to claim 16, wherein thesynthesizer changes the first coefficient linearly from 0 to 1, from lowluminosity to high luminosity, in a predetermined luminosity range, setsthe first coefficient to 0 in a region where the luminosity is lowerthan the range, and sets the first coefficient to 1 in a region wherethe luminosity is higher than the range.
 18. The image processingapparatus according to claim 15, wherein the synthesizer sets the firstcoefficient and the second coefficient used in the dynamic rangeexpansion synthesis to the same value for the plurality of image signalsand the plurality of first signals corresponding to the same pixel inthe image sensor.
 19. The image processing apparatus according to claim15, wherein the plurality of first signals are output for a plurality ofpixels in the image sensor, and the synthesizer sets the secondcoefficient to an average value of a plurality of the first coefficientsused in the dynamic range expansion synthesis of the plurality of imagesignals corresponding to the plurality of pixels.
 20. The imageprocessing apparatus according to claim 19, wherein the plurality offirst signals are signals obtained by amplifying, using a plurality ofdifferent gains, a signal obtained by averaging the partial signalscorresponding to respective pixels in the image sensor for every groupof a plurality of pixels.
 21. The image processing apparatus accordingto claim 19, wherein the plurality of first signals are signals obtainedby amplifying, using a plurality of different gains, the partial signalcorresponding to a single pixel for every group of a plurality of pixelsin the image sensor.
 22. The image processing apparatus according toclaim 15, wherein when the first coefficient is represented by α, thesynthesizer carries out the dynamic range expansion synthesis bymultiplying the image signal amplified at the first gain by (1−α),multiplying the image signal amplified at a second gain lower than thefirst gain by α, and adding the multiplied signals.
 23. The imageprocessing apparatus according to claim 15, further comprising: adetector that detects a focus state on the basis of a phase differencefound using the synthesized first signal and the second signal.
 24. Theimage processing apparatus according to claim 15, further comprising: agenerator that generates a pair of signals for a three-dimensionaldisplay using the synthesized first signal and the second signal.